Skip to content

[AMD][MI355X] Add DSv4 FP4 agentic MTP recipe with DPA tuning - #2254

Open
seungrokj wants to merge 65 commits into
mainfrom
amd/agentx_dsv4_sgl_mtp_0717
Open

[AMD][MI355X] Add DSv4 FP4 agentic MTP recipe with DPA tuning#2254
seungrokj wants to merge 65 commits into
mainfrom
amd/agentx_dsv4_sgl_mtp_0717

Conversation

@seungrokj

Copy link
Copy Markdown
Collaborator

Summary

  • Add new dsv4-fp4-mi355x-sglang-agentic-mtp config (separate from hicache)
  • New benchmark script dsv4_fp4_mi355x_sglang_mtp.sh with EAGLE spec decoding (3 steps, topk 1, 4 draft tokens)
  • DPA tuning: --enable-prefill-delayer, GPU_MAX_HW_QUEUES=5, chunked prefill 8192*TP
  • MAX_RUNNING_REQUESTS = 2*CONC, CUDA_GRAPH_MAX_BS = CONC (cap 128)
  • MEM_FRACTION_STATIC reduced by 0.10 when spec decoding is active
  • Sweep TP8 conc [4,8] non-DPA + conc [16,32,48,52] DPA
  • Image: lmsysorg/sglang-rocm:v0.5.15.post1-rocm720-mi35x-20260714

Test plan

  • Full sweep on MI355X cluster with non-DPA conc [4,8] and DPA conc [16,32,48,52]
  • Verify no HSA out-of-resources crashes during warmup
  • Confirm prefill delayer improves decode throughput at high concurrency

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs


感谢你的贡献!请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人,你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板,包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have

如需进行 PR 验证,请为此 PR 添加 full-sweep-fail-fast 标签(强烈推荐)— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动(flake),重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

@seungrokj seungrokj added AMD full-sweep-fail-fast agentx AgentX benchmarks, recipes, and infrastructure labels Jul 17, 2026
Comment thread perf-changelog.yaml Outdated
Comment thread benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang_mtp.sh Outdated
Comment thread configs/amd-master.yaml
Comment on lines +2468 to +2482
dsv4-fp4-mi355x-sglang-agentic-hicache:
image: lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: cluster:mi355x-amds
precision: fp4
framework: sglang
multinode: false
scenarios:
agentic-coding:
- dram-utilization: 0.80
search-space:
- { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 8] }
- { tp: 8, dp-attn: true, kv-offloading: none, conc-list: [16, 32, 48, 64] }
- { tp: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [64] }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 This PR accidentally leaves a second, byte-for-byte identical dsv4-fp4-mi355x-sglang-agentic-hicache: block in configs/amd-master.yaml (lines 2468-2482), duplicating the pre-existing entry directly above it. It's a copy-paste artifact from using the hicache block as a template for the new dsv4-fp4-mi355x-sglang-agentic-mtp entry — please delete the duplicate block.

Extended reasoning...

The diff adds a second top-level dsv4-fp4-mi355x-sglang-agentic-hicache: mapping key immediately below the pre-existing one, right before the new dsv4-fp4-mi355x-sglang-agentic-mtp: entry that this PR is actually meant to introduce. In the resulting file this shows up as two adjacent, byte-for-byte identical blocks: the pre-existing entry at lines 2452-2466, followed by the newly-added duplicate at lines 2468-2482.

Code path: configs/amd-master.yaml is loaded via yaml.safe_load in load_config_files() (utils/matrix_logic/validation.py:919). PyYAML's default loader does not error on duplicate mapping keys within a single document — it silently keeps only the last occurrence. Since both dsv4-fp4-mi355x-sglang-agentic-hicache blocks are identical, the "last one wins" collapse produces the exact same effective config as if the duplicate had never been added, so there is no behavioral difference today.

Why the repo's own duplicate-key guard doesn't catch it: load_config_files() does have a duplicate-key check at validation.py:930-936, but it only detects a key being duplicated across different config files (it intersects config_data.keys() from the current file against all_config_data.keys() accumulated from previously-loaded files). By the time that check runs, yaml.safe_load has already silently collapsed the intra-file duplicate down to one key, so the duplication within amd-master.yaml itself is invisible to it. There's also no .yamllint config in the repo that would otherwise flag duplicate keys at CI time.

Impact: Purely cosmetic/dead config today, since the two blocks are identical and the parser deterministically resolves to one value. However, this is exactly the kind of artifact that causes real confusion later: if a future PR edits one of the two blocks (e.g. bumping the image tag or tweaking the search space) without noticing the sibling duplicate, the edit could silently have no effect (if the untouched copy happens to load last) or silently take effect while looking like a no-op in a diff review. It's clearly unintentional — the PR description says it's adding only the new -mtp entry, and the mechanism is obviously that the hicache block was copied as a scaffold for the new entry and the copy of the original block was never deleted.

Proof / walkthrough:

  1. Before this PR: dsv4-fp4-mi355x-sglang-agentic-hicache: appears once, at (pre-PR) line ~2465.
  2. This PR's diff inserts 32 new lines directly after it, starting with dsv4-fp4-mi355x-sglang-agentic-hicache: again (identical image, model, scenarios, search-space), followed by the new dsv4-fp4-mi355x-sglang-agentic-mtp: block.
  3. Post-PR, grep -n '^dsv4-fp4-mi355x-sglang-agentic-hicache:' configs/amd-master.yaml returns two hits (lines 2452 and 2468 in the final file).
  4. Running the file through yaml.safe_load collapses these to a single key with the (identical) value of the second occurrence — no functional change today, but a landmine for future edits and a source of reviewer confusion.

Fix: Delete the duplicated dsv4-fp4-mi355x-sglang-agentic-hicache: block introduced by this PR (lines 2468-2482), leaving the original entry and the new dsv4-fp4-mi355x-sglang-agentic-mtp: entry intact.

@github-actions

Copy link
Copy Markdown
Contributor

1 similar comment
@github-actions

Copy link
Copy Markdown
Contributor

@github-actions

Copy link
Copy Markdown
Contributor

@github-actions

Copy link
Copy Markdown
Contributor

@github-actions

Copy link
Copy Markdown
Contributor

@github-actions

Copy link
Copy Markdown
Contributor

@seungrokj
seungrokj force-pushed the amd/agentx_dsv4_sgl_mtp_0717 branch from 5324d24 to 3040b2a Compare July 19, 2026 07:29
@seungrokj seungrokj added evals-only Suppress throughput and run only eval jobs; combine with all-evals to expand selection full-sweep-fail-fast labels Jul 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

@github-actions

Copy link
Copy Markdown
Contributor

@github-actions

Copy link
Copy Markdown
Contributor

@seungrokj seungrokj added full-sweep-enabled and removed full-sweep-fail-fast evals-only Suppress throughput and run only eval jobs; combine with all-evals to expand selection labels Jul 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

…s-python >=3.11) installs on the mi355x sglang-rocm image (default python3 is 3.10)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

seungrokj and others added 2 commits July 31, 2026 16:43
…or prefix-affinity across DP ranks

consistent_hashing keyed on per-request-unique x-correlation-id scattered a
session's turns across DP ranks, so the radix cache could not be reused
(GPU cache hit ~67% vs ~97% theoretical at conc=64). Switch to cache_aware
(radix-tree longest-prefix match per DP rank) and drop the forced
correlation-id routing key so routing follows request content.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

@github-actions

Copy link
Copy Markdown
Contributor

seungrokj and others added 3 commits August 1, 2026 15:32
…0260731 and restore full conc curve

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

1 similar comment
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Long-context agentic warmup (ISL p95 ~433k tokens) with MTP exhausted HBM
(HSA_STATUS_ERROR_OUT_OF_RESOURCES, 0 MB free) in the DSv4 attention indexer's
transient paged-MQA logits buffer, crashing the scheduler. Lower
MEM_FRACTION_STATIC 0.85->0.80 for headroom and cap the prefill chunk at 16384
to bound that per-chunk allocation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

…op prefill cap

Revert the earlier OOM workaround: the passing run 29651998085 (image
v0.5.15.post1-20260714) used mem-fraction 0.90 and a larger prefill chunk yet
did not OOM, so lowering them was not the right lever. Restore mem-fraction 0.9
and remove the 16384 prefill-chunk ceiling.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

seungrokj and others added 2 commits August 2, 2026 12:11
Align the recipe with passing run 29651998085 (v0.5.15.post1) to isolate the
v0.5.16 indexer prefill-workspace OOM: chunked-prefill 4096->8192, swap
--enable-dp-attention-local-control-broadcast back to --enable-prefill-delayer,
router policy cache_aware->consistent_hashing, and restore the SMG routing-key
env. mem-fraction already 0.9.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

seungrokj and others added 2 commits August 2, 2026 12:46
…row to c48

Revert to the last known-good image (v0.5.15.post1-20260714) and run c48 only, to
confirm the OOM regression rides with the v0.5.16 image rather than the recipe.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

…,32,48]

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agentx AgentX benchmarks, recipes, and infrastructure AMD full-sweep-enabled

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

4 participants